Computational Modeling and Experimental Research on Touchscreen Gestures, Audio/Speech Interaction, and Driving

As humans are exposed to rapidly evolving complex systems, there are growing needs for humans and systems to use multiple communication modalities such as auditory, vocal (or speech), gesture, or visual channels; thus, it is important to evaluate multimodal human-machine interactions in multitasking conditions so as to improve human performance and safety. However, traditional methods of evaluating human performance and safety rely on experimental settings using human subjects which require costly and time-consuming efforts to conduct. To minimize the limitations from the use of traditional usability tests, digital human models are often developed and used, and they also help us better understand underlying human mental processes to effectively improve safety and avoid mental overload. In this regard, I have combined computational cognitive modeling and experimental methods to study mental processes and identify differences in human performance/workload in various conditions, through this dissertation research. The computational cognitive models were implemented by extending the Queuing Network-Model Human Processor (QN-MHP) Architecture that enables simulation of human multi-task behaviors and multimodal interactions in human-machine systems. Three experiments were conducted to investigate human behaviors in multimodal and multitasking scenarios, combining the following three specific research aims that are to understand: (1) how humans use their finger movements to input information on touchscreen devices (i.e., touchscreen gestures), (2) how humans use auditory/vocal signals to interact with the machines (i.e., audio/speech interaction), and (3) how humans drive vehicles (i.e., driving controls). Future research applications of computational modeling and experimental research are also discussed. Scientifically, the results of this dissertation research make significant contributions to our better understanding of the nature of touchscreen gestures, audio/speech interaction, and driving controls in human-machine systems and whether they benefit or jeopardize human performance and safety in the multimodal and concurrent task environments. Moreover, in contrast to the previous models for multitasking scenarios mainly focusing on the visual processes, this study develops quantitative models of the combined effects of auditory, tactile, and visual factors on multitasking performance. From the practical impact perspective, the modeling work conducted in this research may help multimodal interface designers minimize the limitations of traditional usability tests and make quick design comparisons, less constrained by other time-consuming factors, such as developing prototypes and running human subjects. Furthermore, the research conducted in this dissertation may help identify which elements in the multimodal and multitasking scenarios increase workload and completion time, which can be used to reduce the number of accidents and injuries caused by distraction.PHDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/143903/1/heejinj_1.pd

Modeling of Stimulus-Response Secondary Tasks with Different Modalities while Driving in a Computational Cognitive Architecture

This paper introduces a computational human performance model based upon the queueing network cognitive architecture to predict driver’s eye glances and workload for four stimulus-response secondary tasks (i.e., auditorymanual, auditory-speech, visual-manual, and visual-speech types) while driving. The model was evaluated with the empirical data from 24 subjects, and the percentage of eyes-off-road time and driver workload generated by the model were similar to the human subject data. Future studies aim to extend the types of voice announcements/commands to enable Human-Machine-Interface (HMI) evaluations with a wider range of usability test for in-vehicle infotainment system developments

Forward collision warning modality and content: a summary of human factors studies

The report summarizes a nonexhaustive sample of 17 studies covering 27 experiments on human factors and forward-collision warnings. Subject samples ranged from 11 to 260 (median=30). Twenty-three experiments were conducted using driving simulators; 4 were on test tracks. Typically subjects followed a lead vehicle that braked abruptly, triggering audio, visual, tactile, or combined warnings. Response/reaction time was reported as a dependent measure in 18 of the 27 experiments, the number of crashes in 8, distance headway (gap) in 3, perceived urgency in 7 (both by the same authors), perceived annoyance in 11, and probability of warning recall in 1. Providing a warning leads to a more desired outcome. Response/reaction times were briefer in 9 of the 9 studies that considered this and all 4 of the studies that examined crashes reported fewer crashes with warnings. Warnings 4 to 10 dB above the background level led to the best performance, but only one study systematically varied warning intensity. Of the combinations explored, multimodal warnings tended to lead to better performance than unimodal warnings, though none of them considered seat-belt-pretensioner activation, an effective way to reduce crash injuries. Studies could be improved by the use of consistent crash scenarios, defined measures, predictions of performance, and including older drivers in test samples.Nissan Technical Center North Americahttp://deepblue.lib.umich.edu/bitstream/2027.42/134038/1/103247.pdf-1Description of 103247.pdf : Final repor

SAE and ISO standards for warnings and other driver interface elements: a summary

This document summarizes 8 SAE documents (4 information reports, 3 recommended practices, and 1 standard), 8 ISO documents (5 standards, 2 technical specifications, and 1 technical report), and 3 NCAP documents. Standards and Recommended Practices describe what must (“shall”) and should be. Information Reports generally provide useful information and guidance without requirements or recommendations. The SAE documents include J2395 (message priority), J2396 (definitions and measures for visual behavior), J2399 (ACC characteristics and user interface), J2400 (FCW operating characteristics and user interface), J2802 (blind spot system operating characteristics and user interface), J2808 (Road/LDW system user interface), J2830 (icon comprehension test), J2831 (recommendations for alphanumeric text messages). The ISO documents include PDTR 12204 (integration of safety warning signals to avoid conflicts), 15005 (dialog management principles and compliance procedures), CD 15006 (specification for auditory information), 15008 (specification and tests for visual information), 16951 (procedure to determine message priority), 17287 (procedure to assess suitability for use while driving), DTS 15007 (measurement of driver visual behavior).Hyundai-Kia America Technical Centerhttp://deepblue.lib.umich.edu/bitstream/2027.42/134039/1/103248.pdf-1Description of 103248.pdf : Final repor

Off-Policy Temporal Difference Learning For Robotics And Autonomous Systems

Reinforcement learning (RL) is a rapidly advancing field with implications in autonomous vehicles, medicine, finance, along with several other applications. Particularly, off-policy temporal difference (TD) learning, a specific type of RL technique, has been widely used in a variety of autonomous tasks. However, there remain significant challenges that must be overcome before it can be successfully applied to various real-world applications. In this thesis, we specifically address several major challenges in off-policy TD learning. In the first part of the thesis, we introduce an efficient method of learning complex stand-up motion of humanoid robots by Q-learning. Standing up after falling is an essential ability for humanoid robots yet it is difficult to learn flexible stand-up motions for various fallen positions due to the complexity of the task. We reduce sample complexity of learning by applying a clustering method and utilizing the bilateral symmetric feature of humanoid robots. The learned policy is demonstrated in both simulation and on a physical robot. The greedy update of Q-learning, however, often causes overoptimism and instability. In the second part of the thesis, we propose a novel Bayesian approach to Q-learning, called ADFQ, which improves the greedy update issues by providing a principled way of updating Q-values based on uncertainty of Q-belief distributions. The algorithm converges to Q-learning as the uncertainty approaches zero, and its efficient computational complexity enables the algorithm to be extended with a neural network. Both ADFQ and its neural network extension outperform their comparing algorithms by improving the estimation bias and converging faster to optimal Q-values. In the last part of the thesis, we apply off-policy TD methods to solve the active information acquisition problem where an autonomous agent is tasked with acquiring information about targets of interests. Off-policy TD learning provides solutions for classical challenges in this problem -- system model dependence and the difficulty of computing information-theoretic cost functions for a long planning horizon. In particular, we introduce a method of learning a unified policy for in-sight tracking, navigation, and exploration. The policy shows robust behavior for tracking agile and anomalous targets with a partially known target model

Connected and Automated Vehicle Based Intersection Maneuver Assist Systems (CAVIMAS) and Their Impact on Driver Behavior, Acceptance, and Safety

Intersection crashes can be potentially mitigated by leveraging deployments of vehicle-to-infrastructure (V2I) and vehicle-to- vehicle (V2V) safety management solutions. However, it is equally critical that these deployments are undertaken in tandem with interventions based on human factors evidence relating to the content and presentation of such solutions. This driving simulator study designed and evaluated a conceptual system - Connected and Automated Vehicle based Intersection Maneuver Assist Systems (CAVIMAS) - aimed at assisting drivers with intersection maneuvers by leveraging connected infrastructure and providing real-time guidance and warnings and active vehicle controls. Results indicate that human factors considerations for the design and deployment of such systems remain paramount, given the findings related to drivers’ trust and acceptance of these systems as measured via surveys and by examining actual driving behaviors.Center for Connected and Automated Transportationhttps://deepblue.lib.umich.edu/bitstream/2027.42/156048/4/Connected_and_Automated_Vehicle_Based_Intersection_Maneuver_Assist_Systems_CAVIMAS.pd

Constructing suffix arrays in linear time

AbstractThe time complexity of suffix tree construction has been shown to be equivalent to that of sorting: O(n) for a constant-size alphabet or an integer alphabet and O(nlogn) for a general alphabet. However, previous algorithms for constructing suffix arrays have the time complexity of O(nlogn) even for a constant-size alphabet.In this paper we present a linear-time algorithm to construct suffix arrays for integer alphabets, which do not use suffix trees as intermediate data structures during its construction. Since the case of a constant-size alphabet can be subsumed in that of an integer alphabet, our result implies that the time complexity of directly constructing suffix arrays matches that of constructing suffix trees

Predicting the metabolic cost of exoskeleton-assisted squatting using foot pressure features and machine learning

Introduction: Recent studies found that wearable exoskeletons can reduce physical effort and fatigue during squatting. In particular, subject-specific assistance helped to significantly reduce physical effort, shown by reduced metabolic cost, using human-in-the-loop optimization of the exoskeleton parameters. However, measuring metabolic cost using respiratory data has limitations, such as long estimation times, presence of noise, and user discomfort. A recent study suggests that foot contact forces can address those challenges and be used as an alternative metric to the metabolic cost to personalize wearable robot assistance during walking.Methods: In this study, we propose that foot center of pressure (CoP) features can be used to estimate the metabolic cost of squatting using a machine learning method. Five subjects’ foot pressure and metabolic cost data were collected as they performed squats with an ankle exoskeleton at different assistance conditions in our prior study. In this study, we extracted statistical features from the CoP squat trajectories and fed them as input to a random forest model, with the metabolic cost as the output.Results: The model predicted the metabolic cost with a mean error of 0.55 W/kg on unseen test data, with a high correlation (r = 0.89, p < 0.01) between the true and predicted cost. The features of the CoP trajectory in the medial-lateral direction of the foot (xCoP), which relate to ankle eversion-inversion, were found to be important and highly correlated with metabolic cost.Conclusion: Our findings indicate that increased ankle eversion (outward roll of the ankle), which reflects a suboptimal squatting strategy, results in higher metabolic cost. Higher ankle eversion has been linked with the etiology of chronic lower limb injuries. Hence, a CoP-based cost function in human-in-the-loop optimization could offer several advantages, such as reduced estimation time, injury risk mitigation, and better user comfort